platform framework #2193
platform framework #2193
Conversation
…onnection b/w frontend and backend; update backend handler for pippy integration
| @@ -157,6 +157,7 @@ def create_mar_file(work_dir, session_mocker, jit_file_path, model_archiver): | |||
| extra_files=os.path.join(EXAMPLE_ROOT_DIR, "index_to_name.json"), | |||
| export_path=work_dir, | |||
| requirements_file=None, | |||
| config_file=None, | |||
There was a problem hiding this comment.
what's the difference btetween config_file and yaml_config_file that shows up in the torchrec test?
There was a problem hiding this comment.
here. the config_file is the model config yaml file. I don't see yaml_config_file in TS code.
ts/model_service_worker.py
Outdated
| @@ -22,7 +21,10 @@ | |||
| DEBUG = False | |||
| BENCHMARK = os.getenv("TS_BENCHMARK") | |||
| BENCHMARK = BENCHMARK in ["True", "true", "TRUE"] | |||
|
|
|||
| LOCAL_RANK = int(os.getenv('LOCAL_RANK', 0)) | |||
There was a problem hiding this comment.
by default i don't believe these environment variables will be set in a single GPU deployment
There was a problem hiding this comment.
Here, this value is used to calculate rpc job's port. When torchrun does not run. 0 is returned so that the normal backend worker is able to get correct port.
| private int parallelLevel = 1; | ||
| private ParallelType parallelType = ParallelType.NONE; | ||
|
|
||
| public static ModelConfig build(Map<String, Object> yamlMap) { |
There was a problem hiding this comment.
This seems like a schema, will this reduce flexibility for users trying to add their own configs are these configs from the YAML that matter to us. Curious why the frontend needs to be aware of of these in the first place
There was a problem hiding this comment.
ModleConfig is the schema for frontend parameters which are completely controlled by TS. Users are only interested in the parameters of handler side. So they are only allowed to extend parameters in backend.
Codecov Report
@@ Coverage Diff @@
## master #2193 +/- ##
==========================================
- Coverage 71.57% 71.44% -0.13%
==========================================
Files 73 73
Lines 3307 3338 +31
Branches 57 57
==========================================
+ Hits 2367 2385 +18
- Misses 937 950 +13
Partials 3 3
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
frontend/archive/src/main/java/org/pytorch/serve/archive/utils/ArchiveUtils.java
Show resolved
Hide resolved
frontend/server/src/main/java/org/pytorch/serve/http/api/rest/InferenceRequestHandler.java
Show resolved
Hide resolved
...nd/server/src/main/java/org/pytorch/serve/http/api/rest/PrometheusMetricsRequestHandler.java
Show resolved
Hide resolved
frontend/server/src/main/java/org/pytorch/serve/wlm/WorkLoadManager.java
Outdated
Show resolved
Hide resolved
frontend/server/src/main/java/org/pytorch/serve/wlm/WorkerLifeCycle.java
Show resolved
Hide resolved
Description
Please read our CONTRIBUTING.md prior to creating your first pull request.
Please include a summary of the feature or issue being fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
It covers the following subtasks in the internal design
-c.Fixes #(issue)
#2192
Type of change
Please delete options that are not relevant.
Feature/Issue validation/testing
Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.
Test A
Logs for Test A
Test B
Logs for Test B
reg.txt
Checklist: